We introduce Dataset Grouper, a library to create large-scale group-structured (e.g., federated) datasets, enabling federated learning simulation at the scale of foundation models. This library facilitates the creation of group-structured versions of existing datasets based on user-specified partitions, and directly leads to a variety of useful heterogeneous datasets that can be plugged into existing software frameworks. Dataset Grouper offers three key advantages. First, it scales to settings where even a single group's dataset is too large to fit in memory. Second, it provides flexibility, both in choosing the base (non-partitioned) dataset and in defining partitions.

Previous works have implicitly studied that FL suffers from the "client-drift" problem, which is caused by the inconsistent optimum across local clients.

However, the cross-client edges naturally exist among clients.

For instance, patients' diagnosis varies substantially across medical centers but collaboratively form long-tailed distributions for

On one hand, the local data of users may contain sensitive information, and privacy-preserving mechanisms are needed.

Federated Learning (FL) allows machine learning models to train locally on individual mobile devices, synchronizing model updates via a shared server.

Despite its pilot progress, personalized federated learning with differential privacy still faces numerous challenges.